Transient fault-tolerance primitives. 
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A million dollar question 


Should the server “die” under the load? 
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Little Little's formula 


L = AW 


e L- number of customers inside (concurrency) 
° À — arrival rate 
° W—time inside 
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A(t) - arrivals 


D(t) - departures 
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A(t) - arrivals 


D(t) - departures 


сопсиггепсу = 
Vr f (Atty-Dtt)) dt 
AIEI > D(t) > O 
A(t) є C 
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Concurrency is not parallelism 


“Concurrency is about dealing with multiple 
things at the same time, parallelism is about 
doing multiple things at the same time” 


Rob Pike 


https://blog.golang.org/concurrency-is-not-parallelism 
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Accumulating сопсуггепсу 
Game 1 


Fair Scheduling 
Workload: 3 


Parallelism: 2 
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Accumulating сопсиггепсу 


Game over! 
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Controlled сопсиггепсу 
Game 2 


Сопсиггепсу Limit = 2 


Queue Size = 2 
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Bulkhead 


const int МахСопсиггепсу з 100. 


SemaphoreSlim bulkhead = new SemaphoreSlim(MaxConcurrency, МахСопсиггепсу); 


//... 
if (lawait bulkhead.WaitAsync(TimeSpan.FromSeconds(1.0))) 


{ 


throw new Exception("Bulkhead rejected"); 


j 


try | await ProcessReguestinternal(); return; | 
finally ( bulkhead.Release(); ) 
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Bulkhead 


маг bulkhead с Policy.Bulkhead(100, 100); 
//... 


bulkhead.Executel...); 
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Bulkhead (Polly library) 


private const int inside - 100; 
private const int outside - 100; 


private static readonly BulkheadPolicy bulkhead - 
Policy.BulkheadAsync(inside, outside); 


ТЕР 
try 


{ 


await bulkhead.ExecuteAsync(...); 


) 
catch (BulkheadRejectedException) |... | 
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100 inside, 500 outside 
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Thread Pool regulates сопсиггепсу 


GC 
pause? 


Thread 


Pool 
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Async restores concurrency 


nc tasks, 
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Retry induced failure 
y м 
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Calculating total time of exponentail 
= ua 4(1-7") 


> ar nim 


k=1 


total = first (2^n - 1) 
first = total / (2^n - 1) 
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Lowering the number ої requests 


Total number of requests lowered v 


Min response time is the same v 
Recovery interval is ok v 
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Сопсиггепсу is lower using 
Exponential Backoff 


О(М log(N) 
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Exponential Backoff 


Policy 
„Handle<HttpReguestException>() 
WaitAndRetry(5, 
retryAttempt => 


TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)); 
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Fixing concurrency spikes 
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Randomization is пої trivial 
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Decorrelated Jitter 


IEnumerable<TimeSpan> DecorrelatedJitter() 


{ 
for (var softAttemptCount - 0.0; softAttemptCount < 4; ) 


{ 
softAttemptCount += r.NextDouble() * 2; 
yield return TimeSpan.FromSeconds( Math.Pow(2, softAttemptCount) * random.NextDouble()); 


j 
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Decorrelated Jitter 


var delay = Backoff.DecorrelatedJitterBackoffV2( 
medianFirstRetryDelay: TimeSpan.FromSeconds(1), 
retryCount: 5); 


var retryPolicy = Policy .Handle«FooException»() .WaitAndRetryAsync(delay); 
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New Polly.Contrib.WaitAndRetry jitter formula - median initial delay 1 second, 5 tries 


Complex policy using Ро 
plex policy using Poliy 
sc.AddHttpClient<IService, Client»(client => | client.Timeout = settings.TimeOutPerReguest; |) 
.AddPolicyHandler( 

Policy 

.TimeoutAsync<HttpResponse Message>(settings.TotalTimeOut)) 
.AddPolicyHandler( 

HttpPolicyExtensions 

.HandleTransientHttpError() 

„Or<TimeoutRejectedException>() 

WaitAndRetryAsync(settings.RetryCount, 

i => TimeSpan 

.FromMilliseconds(20 * Math.Pow(2, 1)))) 
И... 
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Complex policy using ۷ 


И... 
.AddPolicyHandler( 
Policy 
TimeoutAsync«HttpResponseMessage»(settings.TimeOutPerRequest)) 
.AddPolicyHandler( 
HttpPolicyExtensions 
.HandleTransientHttpError() 
.AdvancedCircuitBreakerAsync( 
settings.FailureThreshold, 
settings.SamplingDuration, 
settings. Minimum Throughput, 


settings.DurationOfBreak)); 
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What about а Cirguit Breaker? 
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Cold Start failure 
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Beware - сопсиггепсу accumulates! 
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High contention code 


Threads 


Network 


DB ши Less important operation goes 
first 


Thread is sleeping 
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Fallback for secondary operation 


var fallback = FallbackPolicy<OptionalData> 
„Handle<OperationCancelledException>() 
.FallbackAsync<OptionalData>(OptionalData.Default); 


var optionalDataTask = fallback 
.ExecuteAsync(async () => await CalculateOptionalDataAsync()); 


I.. 
var required = await CalculateRequiredData(); 
var optional = await optionalDataTask; 


var price = CalculatePriceAsync(optional, required); 
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Summary 
e Concurrency needs to be controlled 
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What affects concurrency? 


。 External load 

е Кеўгу 

e Temporary stop 
e Contention 
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Follow the RabbitM0. 


Мы - разработчики в Dodo: 


htt р5 Л. те/а ододеу «blameless culture» + «по 


bullshit». Боремся c англицизмами, 
htt ps ://додо.дем использцем их и снова боремся. 

Создаём систему Dodo 15. Она — 
симбиоз ERP, HRM и САМ-систем. Она 


управляет всем бизнесом Dodo. 


Мы работаем в 15 странах.В планах 


весь мир. Нам нцжен ты. 


Введи, чтобы продолжить: 
> business 

> stack 

> jobs 

> projects 

> test 

> help 
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